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ABSTRACT 

The effects of specific guidelines in the taking and 
rating of tests of language summarizing ability were investigated, as 
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63 native-Hebrew- speaking students f row the Seminar Hakibbuttzim 
Teacher Training College in Tel Aviv. Twenty-six subjects were from 
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were from 2 intermediate EFL classes. Five texts were used in the 
study: two texts in Hebrew and three in English. Two sets of 
instructions were developed: one "guided 14 version with specific 
instructions on how to read the texts and how to write the summaries, 
and the other with the typical "minimal* instructions, scoring keys 
for the texts were based on the summaries of nine Hebrew-speaking and 
nine English-speaking experts. The study demonstrated that whereas 
the guided instructions had a mixed effect on the summarizing of 
native- language texts, they had a somewhat positive effect on 
summarizing of foreign-language texts. In some cases, the guided 
instructions helped respondents find key elements to summarize, but 
in other cases they apparently dissuaded the respondent from 
including important details. Certain ideas seemed to be susceptible 
to rater disagreement more than others. The guidelines and English 
texts are appended, contains 23 references. (Author/LB) 
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ABSTRACT 

The main purpose of the study was to determine the effects 
of specific guidelines in the taking and the rating of tests of 
summarizing ability — tests in which respondents read source 
texts and provide written summaries as a measure of their reading 
comprehension level as well as of their writing ability. The 
study also looked at interrater agreement regarding the rating of 
specific ideas within the summaries. 

The subjects for this study were 63 native-Hebrew-speaking 
students from the Seminar Hakibbutzim Teacher Training College in 
Tel Aviv. Twenty-six were from two high-proficiency English- 
foreign-language (EFL) classes and 37 were from two intermediate 
EFL classes. Four raters assessed the students' summaries in the 
study. The two who rated the Hebrew summaries of the Hebrew 
texts were both native Hebrew speakers. Of the two rating the 
Hebrew summaries of the EFL texts, one was a native Hebrew 
speaker and the other an English speaker. Five texts were 
selected for the study, two in Hebrew and three in English (each 
intermediate and advanced EFL student being asked to summarize 
two). Two sets of instructions were developed, one version was 
"guided" with specific instructions on how to read the texts and 
how to write the summaries. The other version had the typical 
"minimal" instructions. The scoring keys for the texts were 
based on the summaries of nine Hebrew-speaking and nine English- 
speaking experts respectively. All 63 respondents summarized the 
first Hebrew text, 53 summarized the second Hebrew text, and on 
the average, slightly more than a third of the students wrote 
summaries for the EFL texts. 

The study demonstrated that whereas the guided instructions 
had a mixed effects on the summarizing of native-language texts, 
they had a somewhat positive effect on summarizing of foreign- 
language texts. An analysis of the summaries on an idea-by-idea 
basis actually revealed that the guided instructions seemed both 
helpful and detrimental in summarizing of texts. In some cases, 
they assisted respondents in finding the key elements to 
summarize, and in other cases they probably dissuaded the 
respondent from including details that in fact proved to be 
essential in the eyes of the experts upon whom the rating key was 
based. A close-order investigation of consistency across raters 
on an idea-by-idea basis revealed that certain ideas (e.g., 
certain global ideas, as well as matters of detail) seemed to be 
susceptible to disagreement more than others. 
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Introduction 

In order to summarize successfully, respondents need both 
reading and writing skills. First, they must select and utilize 
effectively those reading strategies appropriate for summarizing 
the source text — i.e., identifying topical information, 
distinguishing superordinate from subordinate material, and 
identifying redundant as well as trivial information. Then they 
must perform the appropriate writing tasks to produce a coherent 
text summary — i.e., selecting topical information or generating 
it if none appears explicitly in the text, deleting trivial and 
redundant material, substituting superordinate terms for lists of 
terns or sequences of events, and finally, restating the text so 
that it sounds coherent and polishing it so that it reads 
smoothly (Kintsch & van Dijk 1978, van Dijk 1980, Brown, 
campione, & Day 1981, Brown & Day 1983, Chou Hare & Borchardt 
1984, Basham & Rounds 1986, Davies & Whitney 1984). 

Summarizing tasks on reading comprehension tests have a 
natural appeal as "authentic" tests in this era of communicative 
language testing, given the fact that they attempt to simulate 
real-world tasks in which nonnative readers are actually called 
upon to read and write a summary of the main ideas of a text. 
The results are intended to reflect how the respondents might 
actually perform on such a task. In fact, a real -world summary 
is often quite different from a test summary. Real summaries are 
usually prepared for others who have not read the text and simply 
want to know what it is about. Such readers- would probably not 
be fussy about the form in which the summary appears. Test 
summaries, on the other hand, usually have restrictions as to 
length, format, aiid style, and are prepared for an assessor who 
has already decided what the text is about and wants to see to 
what extent the respondents approximate these prior decisions. 

Thus, the test summary may result in a mismatch whereby 
respondents use one set of criteria in preparing their summaries, 
while the raters use another in assessing them. For example, 
there are undoubtedly differences of perception regarding what a 
"main idea" consists of and the appropriate way to write it up 
(e.g. , precisely how telegraphically). There "nay also be 
differing views as to the acceptability of introducing commentary 
into the summary. Research by Basham (1987) with Alaska Native 
students showed that these respondents used their own world view 
as a filter in the summaries that they wrote, personalizing them. 
If such differences are not eliminated through prior training 
and/or through careful instructions on the test, there could be a 
misfit between the way the summary task is executed and the 
criteria used by the raters to evaluate it. 

Along with possible cultural differences, there are other 
potential causes of discrepancy between the way respondents are 
"supposed" to prepare summaries and the way they actually do it. 
For example, a study of 40 Hebrew-speaking university students 
writing EFL summaries found that while the notes they took on the 
text were of a word-level, "bottom-up" nature, their summaries 
were found to be conducted on a top-down basis — i.e., general 
knowledge. The conclusion drawn by the researchers was that the 
reading was fragmented rather than reflecting ongoing interaction 
with the text that would combine top-down and bottom-up analysis 
(Kozrainsky & Graetz 1986). Likewise, a recent survey of 
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Brazilian studies of reading processes (Cohen 1987) indicated 
that the respondents involved were often not executing 
summarizing tasks in a way consistent with the model of what 
summarizing should entail, as presented above. A study by Holmes 
(1936), for example, found that the six EFL graduate students did 
little monitoring of their summaries. They read in a linear and 
compartmentalized manner, rather than globally so as to extract 
the main ideas. 

Consistent with Holmes' study, a study by Gimenez (1984) 
with five EFL graduate students revealed the major summarizing 
strategy to be that of word-level processing, as opposed to 
syntactic analysis or text-level analysis. Often the summaries 
were found to reflect a focus on only part of the text, with the 
interpretation of the text based on the words that the subject 
had learned (however effectively) and collected up in reading the 
text and on previous knowledge. 

In a study comparing a group of high-proficiency college- 
level EFL students with a low-proficiency group, Johns and Mayes 
(in press) found neither group to be using macro-propositions* in 
their summaries. Furthermore, the low students were found to be 
doing a considerable amount of direct copying of material from 
the source text into their summaries, as it was required that the 
summaries be written in the second language as well. In an 
earlier study, Johns (1985) had found that underprepared natives 
were likewise more prone to use reproductions (copying and 
paraphrase) in their summaries than macro-propositions. In a 
case study, Sarig (1988) helped explain the propensity to lift 
material directly out of the text for use in summaries. She 
found conceptual transformation or reconceptualization at the 
macro-level to be a skill that did not come naturally either in 
native or foreign-language summarizing for a competent college 
student. She concluded that it had to be taught explicitly. 
Nonetheless, the explicit teaching of such reconceptualization 
may not yield such positive results either. Bensoussan and 
Kreindler (1990), for example, found that whereas EFL students 
with a semester's training in summary writing now saw summaries 
as an important tool for grasping the gist of a text, they still 
expressed frustration at their inability to distinguish macro- 
from micro-propositions. 

A previous study by this investigator (Cohen, in press) had 
as its main purpose to investigate the ways in which respondents 
at different proficiency levels carry out summarizing tasks on a 
reading comprehension test. The respondents for that study were 
five native Portuguese speakers who had all recently completed a 
course in English for Academic Purposes (at the Pontificia 
Universidade Catdlica de S«o Paulo, Brazil) with an emphasis on 
reading strategies, including summarizing. They represented 
three proficiency levels. 

The findings showed that the respondents in that study had 
little difficulty identifying topical information, yet they had 
difficulty in distinguishing superordinate , non-redundant 
material from the rest, due in large part to an insufficient 
grasp of foreign-language vocabulary. For their written 
summaries, they did not need to generate topic information 



2 Higher-level or superordinate idea units or statements. 
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because all the texts provided it. They did not, however, have a 
good sense of balance with respect to how much to delete. Either 
they were too vague and general or toe detailed. While there was 
some concern for coherence production, there appeared to be 
relatively little attention paid to producing thoroughly coherent 
and polished summaries. In essence, the respondents in the study 
appeared to be more concerned about their interpretation of the 
source text than they are about their production of a summary. 

Aside from problems that the respondents have in preparing 
summaries of texts on tests, there is the further problem as to 
the reliability of the ratings of these summaries. It has been 
found, for instance, that the statistical results from 
summarizing tasks are not always consistent with results from 
other types of tests (e.g., multiple-choice, short-answer, and 
cloze). Shohamy, for example, set out to compare tests of 
summarizing English-foreign-language (EFL) texts to tests with a 
multiple-choice and an open-ended response format — with 
responses either in native language or foreign language, 
depending on the test version. She found the results from the 
summarizing data so inconsistent with the results on the other 
subtests that she eliminated the findings from the published 
study (Shohamy, Personal Communication, and Shohamy 198 4). 3 

The author's previous study (Cohen, in press) also 
investigated how raters dealt with the responses of the five 
participating students. Two EAP course instructors who typically 
rated the EAP exams of summarizing skill at that institution also 
participated in the study as raters. It was found that there 
were some inconsistencies in the raters' behaviors, underscoring 
the importance of developing rigorous rating keys with main ideas 
and connecting schemata for each text. 

One main purpose of the present study was to determine the 
effects of specific guidelines in the taking, of tests of 
summarizing ability. Tests of summarization do not usually 
include specific, guided instructions for the respondents as to 
how to construct their summary. Previous research on test-taking 
strategies would suggest that instructions on tests may not even 
be read if they are heeded to at all (e.g., Cohen 1984). The 
intention in this study was to build into the test a set of 
instructions that would serve as a genuine guide for the 
respondent. It had been tl'a experience of the investigator and 
of colleagues that test instructions were often notoriously vague, 
and that they presupposed an understanding of how to do the task 
— especially in the case of an activity as complex as 
summarizing. Although it may be thought that respondents would 
draw upon previous knowledge (e.g., from classroom instruction) 
as to how to summarize, it would appear that they do not 
necessarily exercise this knowledge when in a testing situation. 

The other purpose of the study was to investigate the rating 
of such tests of summarizing when a rigorous, empirically-derived 
scoring key was provided, since the absence of such guidelines 



It is also possible that Shohamy's summarization subtest was 
measuring some aspects of reading that the other subtests were not 
e.g., a more overall view of the text. But it was also probably 
measuring writing skill, as well as reflecting possible lack of 
reliability in the ratings. 
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was seen as contributing to a lack of reliability in the ratings 
of summaries. The study took a look at the consistency across 
raters on an idea-by-idea basis in order to determine causes for 
disagreement in ratings. 

The study asked the following questions: 

1. In what ways do guided instructions affect performance 
on a summary task? 

2. How consistent are the ratings of the summaries across 
raters? 

Design of the study 
subjects 

The Respondents ; The subjects for this study were 63 
native-Hebrew-speaking students from the Seminar Hakibbutzim 
Teacher Training College in Tel Aviv. Twenty-six were from two 
high-proficiency English-foreign-language ( EPL) classes and 37 
were from two intermediate EPL classes. 

The Raters : Four raters assessed the students' 
summaries in the study. The two who rated the Hebrew summaries 
of the Hebrew texts were both native Hebrew-speaking 
undergraduates in their last year at the Hebrew University. Of 
the two rating the Hebrew summaries of the EFL texts, one was a 
native-Hebrew-speaking freshman at the Hebrew University and the 
other an English rpeaker doing a graduate degree at Tel-Aviv 
University. The latter was highly proficient in Hebrew, as she 
had received twelve years of schooling at a Hebrew day school in 
the U.S. , was married to an Israeli, and had lived in Israel for 
five years. 

Instruments 

The Texts : Five texts were selected for the study, two 
in Hebrew and three in English. The two Hebrew texts were 
intended to reflect two levels of difficulty — both in terms of 
content, complexity of language, and summarizability (i.e., how 
easy they appeared to summarize). The first Hebrew text (800 
words) was entitled, "Movies — from a Form of Magic to an Art 
Form." It was divided into five sections, each containing a 
subtitle, thus making it !, &ummarizer-f riendly . " The second text 
(1,200 words) was entitled, "Problems with New Israeli Prose — 
Between Separatism and Integration." This text was written in a 
more problematic style, and a successful summary of the text 
required a certain degree of background knowledge about the 
topic. 

Three EFL texts were selected so as to represent three 
levels of difficulty. The easiest (1,000 words), entitled "How 
to Avoid Foolish Opinions," presented a series of procedures for 
avoiding such opinions. Its clear organization facilitated 
summarizing. The second text (1,200 words), entitled "Modern 
Constitutions," was more complex both in language and in its 
conceptual organization. The third article (850 words), entitled 
"Specialization," was intended to be the most difficult article, 
but actually turned out to have a relatively easy structure for 
purposes of summary. 

Instructions for Summarizing: Two sets of instructions 
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were developed. One version was "guided" with specific 
instructions on how to read the texts and how to write the 
summaries. The other version had the typical "minimal 1 * 
instructions. The guided instructions told the respondents to 
read each text in order to identify the most important points — 
those that contained the key sentences in each paragraph or those 
that would make the summary interesting to read. The 
respondents were then instructed to write the summary such that 
the content would be reduced to only the essential points and 
that less important details or those detracting from the main 
points would be eliminated. They were also requested to write 
briefly — e.g., 80-120 words per summary.* Their text was to 
comprise one paragraph with all the ideas linked by connecting 
words. They were requested to write the summaries in their own 
words, and in the case of Hebrew summaries of EPL texts, not to 
translate word for word. They were also asked to write a draft 
first and then to copy it over legibly (see Appendix A) . 

The more traditional, non-guided instructions simply told 
the respondents to read each text so as to be able to write a 
summary of it. They were asked to be brief — 80-120-word 
summaries, to write their summary first in draft form and then to 
copy it over legibly. 

Construction of Rating Keys: The two Hebrew texts were 
read and summarized by nine Hebrew speakers, all experts in the 
areas of reading and writing. Three were university lecturers, 
specializing in discourse. The rest were university students of 
language arts and teachers in their own right. The three EFL 
texts were read and summarized in English by nine native English- 
speakers, most of whom were university instructors of EFL. The 
summaries of these experts vere analyzed and a key was 
constructed to include only those main ideas and linking ideas 
that a majority (i.e., five or more) of the experts had included 
in their summaries (after Sarig 1989). In this study, no effort 
was made to distinguish the macro- from the micro-propositions in 
the scoring key. 

The key for the Hebrew texts was produced by one of the two 
raters. The key for the EFL texts was produced by the 
investigator and then translated into Hebrew by the Hebrew- 
speaking rater of the EFL texts, whose English skills were also 
quite advanced. The EFL key was translated into Hebrew. The 
keys appeared in the form of a list of numbered ideas, each in 
sentence form. Table 1 shows the level of agreement for each 
point selected per passage. 

The summaries of the Hebrew experts reflected an 80% average 
agreement as to which main ideas and connecting ideas should be 
included in the summary. The summaries of the EFL experts 
reflected an 85% average agreement as to which ideas to include. 
Hence, it should be noted that there was not full agreement even 
among experts as to which ideas were essential to the 
construction of a meaningful summary. 



*The stipulation of number of words is based on common practice in the 
United States. In Israel, students are used to being told to write a 
paragraph or a page (a less precise measure) . 
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Data Collection Procedures 

The students were requested to write the summaries in two 
meetings of their EFL course — the first one for the Hebrew 
texts, the second one for the English texts. The sessions took 
place in December, 1988, and were one week apart. Each lasted 
approximately 1^ hours. The task with guided instructions was 
administered to every other student according to how they were 
seated in a given classroom. The other students received tht* 
unguided version. All students received in the second sitt?f:7 
the same type of instructions as they had received in the first, 
as this time their name was written on the instruction sheet in 
advance. The tasks were not presented as obligatory and were not 
to count towards the students' grade in the course. 

All 63 respondents summarized the first Hebrew text, 32 
receiving guided instructions and 31 not. Fifty-three 
respondents summarized the second Hebrew text, 27 receiving 
instructions and 26 not. On the average, only a little more than 
a third of the students wrote summaries for the EFL texts. A 
breakdown of the numbers of respondents for each text according 
to version (guided vs. unguided) and respondents' level of 
proficiency (intermediate vs. advanced) appears in Table 2. The 
15% of the students who chose not to summarize the second Hebrew 
text and the 65% who chose not to summarize the EFL texts did so 
primarily because they found those texts too difficult to 
summarize. 

Although the instructions for both the guided and unguided 
versions specified that the students were to write a draft 
summary and then copy it over neatly, only seven of the 67 
students did this for the Hebrew summaries, four of the 
intermediate students and three of the advanced ones. As for the 
summaries of the EFL texts, only one advanced student wrote a 
draft, as that student had done for the Hebrew texts as well. 
Thus, this element in the instructions was not followed in either 
of the two versions. 

Data Analysis Procedures 

The raters received brief training by the investigator in 
rating the summaries and then did several of the ratings in the 
presence of the investigator in order to resolve problems of 
immediate concern. The raters were also asked to make note of 
problems to be resolved in consultation with the investigator 
after completing the ratings. Each main idea and linking idea 
received one point in the rating process. No effort was made to 
have the pairs of raters of the Hebrew and EFL texts "conference" 
together while learning how to do the ratings. However, ratings 
of the two pairs of raters for the Hebrew and EFL texts were 
correlated to determine interrater reliability and discrepancies 
were identified on a point by point basis in order to determine 
the types of ideas for which raters had difficulty reaching 
consensus . 

Crcsstabulations were run using the SPSS-PC program to check 
for differences in success at summarizing between the respondents 
given the guided and unguided versions. The Correlations program 
was run to determine interrater reliability. 

9 
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Findings 

1. The Effect of Guided Instructions on Summarizing 

a. The Summaries of Hebrew Texts; Guided instructions 
seemed to have a mixed, or even negative effect on the 
summarizing of native-language texts. On the first Hebrew text, 
the one an movies, the second rater rated the unguided group 
significantly better overall than the group with guided 
instructions. Yet when behavior was examined at the idea-by-idea 
level, some interesting differences emerged suggesting that the 
provision of guided instructions has a differential effect on 
summarizing, perhaps helping in some instances and interfering in 
others. In examining the main ideas within that Movie text, item 
#5 was one for which the unguided group were rated significantly 
better by both raters (see Table 3). This idea involved giving 
details as to why movies are an art form (e.g., they deal with 
shape, color, movement, words, and music). It is likely that 
those getting the instructions warning them not to include 
unnecessary details were reluctant to provide this level of 
detail for fear that it would be rated as extraneous. 

On the second Hebrew text there were two significant 
differences by version, one favoring the guided and one favoring 
the unguided instructions. In the first instance, the guided 
group which was told to identify all the main ideas in the key 
sentences included idea #6 more than did the unguided group. 
This idea consisted of the second of two examples of "separatism 
and then reintegration in Israeli prose." In other words, the 
guided group was more sensitive to including both of the 
examples. In the second instance, the unguided group was more 
likely to include in their summary a linking statement to the 
effect that new Israeli prose is characterized by continuity in 
the midst of apparent separatism (see Table 3). In this case, it 
was the guided group that was reminded cf the importance of 
linking up ideas and yet the unguided group practiced it more 
successfully in this case. 

b. The Summaries of EFL Texts ! In the EFL text 
summaries, there were two instances of significant difference by 
version, according to the first rater, both in favor of the 
guided group. The guided group were more likely to include the 
first idea in the summary of the Modern Constitutions text — 
namely, the idea which provides an historical perspective for the 
passage. In addition, they were also more likely to include the 
detail that countries differ as to the number of special checks 
and balances stipulated by their constitution (see Table 4). 
Hence, with respect to native-language summaries of foreign- 
language texts, the group given the fuller instructions seemed 
more sensitive to include the introductory idea as well as to 
include detail that might not be deemed central to the summary. 

2. The Consistency of Ratings Across Raters 

a. The Raters of th e Hebrew Texts : The average 
correlation of the ratings of the two Hebrew native speakers 
across the two texts that they rated, was significant but 
relatively low as a reliability coefficient (r - .73, p<.O0l). 
On the Movie text, there was significant agreement on five of the 

_ 10 _ _ _ 
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points, four involving basic description and one involving basic 
exemplification (i.e., what makes movies an art form) (Table 5). 
There was inconsistency on three points, two involving contrast 
(both involving the contrast between then and now) and one 
involving detailed exemplification (i.e., regarding modern 
technology in moving making) . 

On the New Israeli Prose text, there was considerable 
agreement involving the opening contrastive idea (i.e., 
separatism and integration), two examples of this, and a linking 
idea (i.e., dealing with the continuity in Israeli prose) (Table 
5). There was some agreement concerning a descriptive point 
regarding the origins of separatism, and less agreement regarding 
two other descriptive points — a point about continuity and a 
point introducing the two examples of separatism and integration. 

b. The Raters of the EFL Texts ; The average 
correlation of the two raters across the th: s»e EFL texts was high 
(r ■ .85, p<.001), especially considering that the first was a 
native Hebrew speaker and the second a native English speaker. 
Yet when we look at the level of consistency from idea to idea 
within the three EFL texts, we note that there are marked 
differences (Table 6). On the Opinions text, the raters agreed 
on their ratings of the fifth and sixth ideas, and more or less 
so on the third idea, but not for the first or opening statement 
(i.e., the article suggests procedures for avoiding foolish 
opinions), the second, or the fourth (an extension of the second 
suggestion) . 

On the second text, the raters agreed on the opening 
statement which gives an historical perspective and on three of 
the basic points (i.e., a justification for constitutions being 
above law, protection of special communities, constitutions of 
countries differ), but disagreed on the topic sentence linking 
the passage together cohesively (in terms of- the constitution as 
a fresh start) and on an item dealing with details of what rights 
a constitution includes. On the third text, the raters agreed on 
all but the opening idea (i.e., that by the end of the nineteenth 
century the intellectual general ists had given way to 
specialists) (Table 6). 

Thus, the raters were inconsistent in their ratings of 
several of the more global, linking ideas and of ideas involving 
details in the EFL texts. T^iese discrepancies would not appear 
to be the result of native language differences. Instead, it 
would seem that these are perhaps the ideas that lend themselves 
to the most controversy in rating, even when a precise key is 
provided. This study did not attempt to investigate why raters 
disagreed on their assessments of respondents' answers as 
conducted in a previous study (Cohen, in press), but the findings 
would indicate that certain ideas on a rating key are problematic 
for scoring and others are not, and that the language of the 
rater may not be as important here as other factors. 

Discussion and Conclusions 

The current study demonstrated that whereas guided 
instructions had a mixed effect on summarizing of native-language 
texts, they had a somewhat positive effect on summarizing of 
foreign-language texts. An analysis of summaries on an item-by- 
itera basis revealed that, in reality, the guided instructions 

U 
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appeared to be both helpful and detrimental. In some cases, they 
assisted respondents in finding the key elements to summarize, 
and in other cases they probably dissuaded the respondent from 
including details that in fact proved to be essential in Uie eyes 
of the experts upon whom the rating key was based. 

Since all the students were admonished to read the 
instructions carefully, it is likely that they did so and that 
the results do reflect the two conditions for test taking — with 
and without guided instructions. The results, then, would 
suggest that the nature of the instructions in foreign-language 
testing may be of more importance than in native-language 
testing. In other words, test constructors may wish to give more 
explicit guidance to nonnatives concerning recommended means for 
carrying out the given task than they would give to natives. 
With regard to native-language testing tasks, it is likely that 
the respondents may either pay less attention to the instructions 
altogether or may simply exercise appropriate test-taking 
strategies without needing a reminder as to how to do it. 

With respect to interrater consistency, it was found that 
the raters differed in their ratings both of several of the more 
macro-level, global, linking ideas, as well as of micro-level 
ideas involving details in the EFL texts. It would appear that 
certain ideas at these two disparate levels lend themselves to 
controversy in rating, even when a precise key is provided. 

Limitations in the current study can also be seen as 
recommendations for future research: 

(1) Although the intention was to indicate for each summary 
who the text was being summarized for, this was inadvertently 
left out of the instructions for both groups. If we take the EFL 
text on Specialization, for example, it was intended that the 
respondents would be told, say, that they work for a company that 
makes documentary movies and that their boss, has asked them to 
read an article on the dangers of overspecialization and to 
summarize it for him in order that they may get some ideas for 
the preparation of the script for the documentary. Perhaps the 
addition of this modicum of functionality would have prompted 
more of the students to do the task in a context where 
participation was not obligatory. 

(2) It may have been beneficial to the research to have the 
summaries count for course credit. Then perhaps more of the 
students would made an effort to summarize the second Hebrew text 
and the two EFL texts that they were assigned. The fact is that 
the respondents probably did not have behaved on these tasks as 
th£y would on a genuine test. 

(3) The first rater of the Hebrew texts had an observation 
regarding the scoring key. She felt that by using a key based on 
the judgment of "experts," it was skewing the assessment away 
from the level of the students who were being assessed. She felt 
that the experts wrote a different type of summary from the 
seminar students — i.e., that they used more of a logical 
sequence in the presentation of their ideas, demonstrated a 
tighter use of words, and displayed greater overall quality in 
their writing. Besides for these observations, she noted that 
whereas there were points that the experts did not agree upon and 
were thus left out of the key, nonetheless students felt these 
points to be important enough to include. A compromise would be 
to build a rating key based both on the suggestions of the 
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respondents as to key ideas and the insights of the examiners, as 
was done by Bensoussan and Kreindler ( 1990 ) . 

In the work by Bensoussan and Kreindler, as soon as the 
respondents finished summarizing the given texts, they were 
presented with the two teachers' set of main ideas for the 
summary arrived at through conferencing, and were asked as a 
group to react to that set. If they disagreed with any of the 
teachers' points, they had to convince the teachers that a change 
was in order in the scoring key. Apparently the students became 
more proficient at this task as they performed it various times. 
While at the beginning of the course the teachers dictated the 
correct summaries, by the end of the semester, the students had 
learned how to suggest changes (Bensoussan, Personal 
Communication) . 

(4) No test-taking data were collected in this study as had 
been collected using verbal report techniques in studies r^orted 
on elsewhere (e.g., Cohen 1984, in press). Hence, there was no 
assurance, for example, that the students taking the guided 
version had paid full attention to the elaborated instructions 
that they received. It was impossible to read those instructions 
aloud since every other student received the unguided 
instructions. Perhaps a future study would collect data 
regarding the processing of the instructions. 

(5) The raters did not conference with each other 
concerning the appropriate ratings far each summary. A follow-up 
study would include rater conferencing with other raters, say in 
pairs, in order to allow for potentially greater consistency 
across raters. Such conferencing could also be studied at the 
process level through analysis of verbal reports in order to 
determine what such conferencing entails. 

(6) More global ideas were not given more credit in the 
summaries than more local ideas. Future work could distinguish 
main ideas at the level of micro- and macro-propositions to see 
the extent to which the level of abstraction of the proposition 
influences the behavior of the rater. 
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Table 1 

Level of Agreement for Points Selected for Key 

Hebrew Texts: "Movies — from a Form of Magic to an Art Form" 
1. 9/9 2. 6/9 3. 7/9 4 . 6/9 5 . 5/9 6. 6/9 7. 8/9 8. 7/9 

"Problems with New Israeli Prose — Between Separatism and 
Integration" 

1. 8/9 2. 7/9 3. 7/9 4. 6/9 5. 9/9 6. 9/9 7. 8/9 
"How to Avoid Foolish Opinions" 

1. 8/9 2. 8/9 3. 7/9 4. 9/9 5. 9/9 6. 8/9 7. 8/9 
"Modern Constitutions" 

1. 8/9 2. 9/9 3. 6/9 4. 8/9 5. 7/9 6. 5/9 7. 8/9 
"Specialization" 

1. 7/9 2. 5/9 3. 9/9 4. 9/9 



Table 2 

Respondents for Text Summaries 



EFL Level: 
Hebrew Texts: 

Movies as an Art Form 19 
New Israeli Prose 16 



EFL Texts: 

Foolish Opinions 
Modern Constitution 
Specialization 



Guided Version 



Int. Adv. Total 



12 
8 



13 
11 



3 

3 



32 

27 

12 
11 
3 



Unguided Version 
Int. Adv. Total 

. 18 13 31 
17 9 26 



11 



Overall 



63 
53 

17 
22 
10 



14 



Cohen - 12 



Table 3 

Summaries of Hebrew Texts by Version 
(with/without Guided Instructions) 



Rater 2 - Total Perf. 
Version 
Points Guided Unguided 





0 








Movies 


2 




4 


1 


as 


3 




13 


2 


Art Form 


. 4 




8 


18 


Text 


5 




4 


7 


in 


6 




2 


1 


Hebrew 


7 




1 


2 






N = 


32 


31 






d.f 


. 5 





Rater 



Pts 
0 
1 



1 - Idea 5 Rater 
(Details) 
Version 

Guided Ung. Pts 

26 15 0 

6 16 1 



2 - Idea 5 
(Details) 
Version 
Guided Ung. 
28 18 
4 13 



d.f, 



d.f 



Chi-Square=15.19** Chi-Square=7 . 48** Chi-Square=6 . 92** 





Rater 1 - Idea 6 


Rater 2 - Idea 7 




(Second Example) 


(Linking Idea) 




Version 


Version 




Pts. Guided Ung. 


Pts. Guided Ung. 


New 


0 7 14 


0 15 7 


Israeli 


1 20 12 


1 12 19 


Prose 






Text 


N = 27 26 




in 


d.f. 1 


d.f. 1 


Hebrew 


Chi-Square=4 . 32* 


Chi-Square=4 .47* 




Table 4 






Summaries of EFL Texts by Version 




(with/without Guided Instructions) 




Rater 1 - Idea 1 


Rater 1 - Idea 6 




(Perspective) 


(Detail) 




Version 


Version 




Pts. Guided Ung. 


Pts. Guided Ung. 


Modern 


0 0 5 


0 6 10 


Consti- 


1 11 6 


1 5 1 


tution 






Text 


N = 11 11 




in 


d.f. 1 


d.f. 1 


EPL 


Chi-Square=6 . 47** 


Chi-Square=3 . 67* 



** p<.01 
* p<.05 



Table 5 



ERIC 



15 



Cohen - 13 



Interrater Agreement on the Hebrew Texts 



Idea: 1 



7 8 Overall 



Movie 
Text 



De- Descr , 
scription 

.55*** .84*** 

N=63 



Con- Descr. Basic Descr. De- Con- 
trast Exempli- tailed trast 

fication Exemplif. 
.23 .59*** .83*** .57*** .10 .20 .34* 



Idea : 1 



Overall 



Prose 
Text 



Contrast De- Descr. Descr. Exemp- Exemp. Link 

scription lification 
.60** .45** .04 .06 .89*** .88*** .65*** .81*** 

N=53 



Average correlation of ratings across both texts = .73*** (N=53) 



Opin- 
ions 
Text 



Table 6 

Interrater Agreement on the EFL Texts 

Idea: 1 2 3 4 5 6 7 

Purpose 1st 2nd Ext. 3rd 4th 

for Sugges- Sug. of Sug. Sug. 
Article tion 2nd Sug. 

.19 .04 .45 -.09 .54* .75*** 



Consti- 
tution 
Text 



N=16 

Idea : 1 2 
Perspec- 
tive 

.59** — 



N=22 



3 4 5 6 7 

Justi- De- Descrip- Descr. Link 
fication tails tion 

.47 .20 .91** .46 -.05 



Overall 



.56* 



Overall 



.62** 



Idea: 12 3 4 Overall 

Opener General Ignor. Result 

Ignorance in Field of Mechanization 
Special- .17 1.00** .47 .73 .94*** 

ization 
Text 

N=7 



Average correlation of ratings across all three texts = .85*** (N=27) 



* p<.05 
** p<-01 
*** p<.001 
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Appendix A 

Translation of the Elaborated instructions: 

Reading Text and Writing a Summary of the Text 

Before you are four texts, two in Hebrew and two in English. 

Instructin ns for Reading 

Read each text so as to extract the most important points from 
it, that is, those points that contain the key sentences (for the 
given paragraph) ; or those points that the reader of the summary 
will be interested in reading. 

instructions for Writing t he Summary fin Hebrew) 

1. Reduce the information to main points only: avoid the 
inclusion of redundant information. Including ^his information 
will detract from your score. 

2. Write briefly: the length of the summary is to be 80 words 
for the first passage and 120 words for the second. 

3. Write the summary as a single passage: use connecting words 
to link the points together. 

4. Do not translate literal}" • write the summary in your own 
words . 

5. Write a draft first, anc*. then copy it over legibly. 

Translation of the Traditional Brief Instructions: 
Reading the Text an d Writing a Summary of It 

Before you are four texts, two in Hebrew and two in English. You 
are to read each one so as to write a summary of it. 

instructions for W riting the Summary fin Hebrew), 

1. Write briefly: the length of the summary is to be 80 words 
for the first passage and 120 words for the second. 

2. Write a draft first, and then copy it over legibly. 
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APPENDIX B 



English Translation of Hebrew Texts and Scoring Keys 



MOVIES — FROM A FORM OF MAGIC TO AN ART FORM 

by Lea Dovev-Rosenbaum 

Films as a Form of Magic 

The first movie makers and viewers did not see movies as an 
art form, but rather as amusing trickery. The first creations 
were the property of eccentrics and fanatics. They claimed that 
just as it was possible to film something at rest, so it was 
possible to film motion by combining a series of single shots and 
then projecting them rapidly one after another. The idea seemed 
amusing at the time, and this is what the audience thought when 
they came to see the first film shows. The spectators were 
common people: the unemployed and the illiterate. This audience 
was not at all familiar with works of art, were not educated to 
appreciate art, and were preoccupied with the daily concerns of 
making a living. They simply wanted to be entertained, and the 
new amusing form of trickery served their needs. They thoroughly 
enjoyed the hocus-pocus of the films. The first films startled 
and marveled the viewers. They did not understand how it was 
possible for a train to travel on a stretched out cloth and how 
people were able to walk on that same cloth. Their understanding 
was that films were an illusion of reality that through a miracle 
were transferred onto the screen. It was as much a marvel to 
them as a magician's pulling rabbits and doves out of a hat. 

Films at Present 

Although films were not originally created to be an art 
form, they have become the principle form of art that we come in 
contact with in our daily lives. Films were not created as an 
art form — to say that seems strange today, as millions of 
people are not interested in art of any kind except for movies 
and TV. Furthermore, it is difficult to imagine how painting and 
sculpture, literature and music of today would appear if movies 
had not changed our concept of art so greatly. Artists of today 
paint, sculpt, write literature, stage theatrical productions, or 
create any other works under the influence of films, which make 
use of special perceptual senses. 

From a Form of Magic to an Art Form 

Technological advances and sophisticated equipment changed 
the film industry from an amusing form of magic to an art form. 
The new techniques were able to let the film industry overcome 
obstacles that constrained other forms of art, to such a degree 
that they saw in the film industry possibilities greater than 
those in any other art form. Every art form has its limits, if 
you are a painter, you cannot use words. If you are a sculptor, 
you can not expect the sculpture to dance. Painters have their 
paint and a canvas, which is a lot but also limiting.. The gap < 
between reality and art is always there because artistic means ' 
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are limited, while reality is infinite. The invention of movies 
was a challenging step towards closing the gap between art and 
life. Movie making is the art of all art forms, it contains 
everything: shape, color, movement, words, and music. It is no 
wonder that movies appear so complete, rich and full of 
opportunities, more than any other art form. Through the use of 
the movie camera, films can show wide-open spaces, and can 
realistically recreate the different scenes that people see 
walking along the streets, driving in their cars, etc. Unlike 
the human eye that sees everything from the same angle, movie 
makers can change the angle of filming and the distance of the 
camera. They can decide the amount of light and sharpness of 
focus. All of these can bring about ama2ing changes and reveal 
new scenes for the viewers. 

Are movies an art form? 

The fact that movies can create an illusion of reality 
better than any other art form brought up the question: are 
movies an art form? The issue facing the spectators at the turn 
of the century was: is it good that filmmaking has greater means 
for creating an illusion of reality? How can films be thought of 
as art if they are so similar to reality? If the filmmaker does 
not in any way process the pictures that are caught by the eye 
but just films the scenes — then this is just a simple, 
mechanical craft, and for this you do not need any talent. The 
filmmakers' reply was that just like any other art form, movies 
have possibilities and boundaries of their own. The film 
industry can be similar to reality, more than any other field, 
but it can also depart from reality. Film in the hands of the 
filmmaker is like clay in the hands of sculpture. Photographer 
decide what to film, how to film, what to add, and what to 
delete. They can edit the film in a way that will create a new 
reality, that is not just a carbon copy of existing reality. 
They can create a reality that is worked, sophisticated, complex, 
pretty, and even more interesting than reality. Filmmakers, the 
pioneers of filmmaking, wanted to create films that would not 
just be a substitution for reality, but which would create an 
artistic experience. 

Artistic Means for Movie Making 

For filmmakers there are technical means whereby they can 
change reality. What can the filmmaker do to create films in the 
modern world, different from reality, deeper, and more 
interesting? One of the principle means of design that stand 
before the filmmaker is related to the special possibilities of 
the camera and its limits. The constraint of the camera is that 
its range of focus is more limited than that of the human eye. 
The camera can film just what fits within a given frame. This 
frame is rectangular and its measurements are standard. But this 
constraint can also be a great advantage. The filmmaker can 
decide which details from the enormous range of data will be 
included inside the frame and which will not. They decide upon a 
certain portion of reality and the one that is chosen is not 
accidental. The limited range of vision of the camera gives 
filmmakers the ability to focus the viewers' attention on certain 
pretty or meaningful details They may have an event happen 
beyond the limits of the frame and may select the most important 
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event at that moment in order to heighten suspense. (For 
example, the filmmakers may decide not to let us see a love scene 
in a room but instead have us look at a view that the lovers see 
from the window.) Another advantage that the camera has unlike 
the human eye is. that it can give different perspectives from 
different distances, different viewpoints, at with different 
intensity of lighting, at different degrees of focus, etc. 

Another interesting aspect that without it, filmmaking would 
be a boring copy of reality, is editing. Editing a movie, i.e., 
cutting out parts of the film and reconnecting them elsewhere in 
a different order, this is where the most important difference is 
between the human eye and the perspective of the movie maker. 
The human vision is always continuous in time and space, and 
there are no "jumps": we see just what is in our field of 
vision. The view of the filmmaker is different. Using editing, 
the filmmaker can decide which scenes we will see and which we 
will not, and in which order. The possibility standing in front 
of the editor are different and manifold. 



Scoring Key for "Movies: From a Form of Magic to an Art Form 

1. Movies started as entertainment, a circus act, amusement. 

2. The spectators that came to the movies were simple folks, 
from a low socioeconomic level, without an art background. 

3. Movies today have turned into the principle art form, the art 
form of all art forms. 

4. This art form modifies or influences all other art forms. 

5. It includes the following components: shape, color, movement, 
words, and music. 

6. Thanks to modern technology, the film industry as changed 
from simple amusement into an art form. 

7. Through the artistic channels of movie making, i.e., choice 
of subject matter, focus, and editing, reality is fashioned into 
a new creative art form. 

8. The contrast between the notion of early films as simple 
entertainment and current films as an art form. 



PROBLEMS WITH NEW ISRAELI .PROSE: 
BETWEEN ISOLATION AND INTEGRATION 

by Benjamin Yitzhak Michael i 

In the new Hebrew prose originating in Israel, two markedly 
different trends emerged that were not simultaneous but appeared 
one after the other: separatism from literary heritage at the 
start, during its period of fermentation and propagation, and 
later on in its maturation, an effort to overcome this 
separatism, out of a recognition of the power of continuity. The 
nature of this new prose, whose first buds were seen during the 
struggle with an occupying power, and whose fruits ripened during 
the years of the State, is perceived retrospectively not as a 
foreign implant, whereby its winds brought its seod from a 
foreign land, but rather as a Hebrew "tree of creation" from 
whose roots it emerged and from whose juices it was saturated. 
Its buds, and probably its unripe fruit, emphasized its 
difference and individuality. Here comes its ripe fruit and 
indicates to us the similar and closeness, the fundamental unity 
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start in the Days of October or during the period of the struggle 
that led to it — that is to say, a brief chronological affinity 
or a brief thematic affinity. Nevertheless, the conscious 
separatism, militant in nature and abominably cruel, was to no 
avail. Underneath the conscious stand, however, latent, 
irrational factors were operating, and at the first opportunity, 
these surfaced. Whatever the Revolution did not accomplish in 
bringing about ideological division along class lines within the 
Jewish collective, was brought about by the massive destruction 
that erased class distinctions and brought together all the 
different divisions of the Jewish people under the tyranny of the 
Nazis. A brotherly hand was extended to groups of Jews outside 
the country of the Revolution. According to Bergson, the ancient 
cry could be heard, as in his play, "I won't die for I will 
live!" Poets and storytellers who were formerly content to 
praise Stalin and events taking place in the soviet Union, were 
reminded of their Jewish past and gave expression to their 
excitement in literature, revolving around ancient figures such 
as Rachel, King Solomon, Isaiah, the Maccabees, Bar Kochba, Rabbi 
Akiva, and Yehuda Halevy. Suddenly with the destruction of the 
Jews in the Holocaust, separatism vanished. There was a 
continual undercurrent of forces seeking a way of breaking 
through the man-made barriers. In other words, as long as it had 
not been accomplished through assimilation — which takes many 
generations — conscious separatism was eliminated by reality 
itself. 

The second example: the "Canaanite" group. Certainly this 
group and the Yevreyskaya were not on one plane and did not 
aspire to the same goals. The latter group denied its heritage, 
seeking to eliminate any traces of identification with the Jewish 
people, and preserving what was merely subsidiary, the Yiddish 
language that was spoken by the Jewish community at large in the 
Soviet Union; whereas the former group adhered to its historic 
homeland declaring its desire to extend its borders, at the same 
time that it was planning to reduce the geographic borders of the 
Jewish nation. That is, their view was that Jews of the Diaspora 
were second-rate and should not have a stake in Eret2-Israel and 
in its people. Their use of historical roots produced a strange 
schism: the "Canaanites" acknowledged their historical homeland, 
but thfty denied the reality of the historical people, its 
generations, and its dispersion. Moreover, this inconsistency 
with regard to facts of Jewish history does not just concern the 
Diaspora, but also applies to the homeland as well. Not for 
naught were they drawn to the long lost period of their ancestry: 
before we became a people, before the Lord gave us the Torah and 
the Ten Commandments, before the appearance of the prophets, 
whereas the later period was perceived for all that primordial 
reminiscing as memories of the Molech, Baal, and Astarte. It was 
not that those ancient times were enlivened to ensure a foothold 
in later times, but that they, and they only, were considered to 
be the be-all and end-all of everything. In this way the 
"Canaanites" transformed the Israeli nation from an organic 
being, within which unity operates in spite of the multitude of 
changes, into a mechanical being, which is like clay in the 
potter's hands. But such a change is sure to be undone. Just as 
a living tree does not exist only as roots at the bottom and 
leaves at the top, but rather also as a trunk and the ever 
increasing network of branches, so too a change does not consist 
solely of a beginning and an end with a void between them. 
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But is not this Canaanitism but an anomalous phenomenon? 
Was it only an arbitrary occurrence? Whoever does not feel 
content with its public declarations but rather looks for a 
hidden motive will not rush to such a conclusion. When looking 
for traces of the Canaanites, one will find evidence of their 
predecessors and then will discover that the younger generation 
of authors was not that different, in fact:, from the older. This 
did not imply departing for ancient regions, as did the authors 
of the Enlightenment Movement, but to a faithfulness to their 
intentions. Michal was fascinated by the love of Solomon and the 
wisdom of Ecclesiastes while perceiving what was tragic in the 
extinction of beauty and grandeur; Kappu envisioned an ancient 
period when the Jewish people were still rooted in their 
homeland, in contrast to their current detachment — as an 
expression of their hope of restoration; Yalag employed the 
clashes between King Zedekiah and Jeremiah as a means of 
indicating his preference for statehood and the need for heeding 
the words of the prophets. What these authors had in common was 
a nostalgia for the period of independence of the people of 
Israel while prospering in their homeland and subordinated to the 
regime of the high priests, judges, and kings. Their followers 
differed in that they aimed for an earlier period — that of the 
pagans, before these impulses were restrained and subjected to 
the Bible, the Ten Commandments, and public governmental 
structures. This was the case with Frishman in his "desert" 
tales (referring to the Book of Numbers) and Berdichevsky 
exploring the manifestations of the rule of blood and impulse; 
the same was true of Tchernichovsky in his psalms to Astarte, 
Baal, and Shneor in the hidden tablets (of the decalogue). Of 
course, there were differences among these later authors, but 
what they had in common was their fascination with the primitive, 
a fascination to account for their repulsion- of the symptoms of 
decadence exhibited by the Jewish people oppressed in their 
exile. These authors were united by a wish to cure the Jewish 
people through reigniting their impulses that were suppressed, 
and transfusing healthy blood into its stricken body — to bring 
life into the nation. Apparently, the major difference between 
them and the Canaanite sons was how they dealt with a change of 
circumstances for the Jewish people. The former were content in 
a nostalgic visionary form, matching the diasporic detachment in 
exile and the yearning to overcome it. Jhe latter group, on the 
other hand, aspired for political-social concretization, that 
would match the reality of the homeland. Still wasn't there a 
hidden inclination for a reunion in the claim for separatism? 
Was it not a meritorious continuity achieved through a wrongful 
separation? That is to say, it was an intermittent continuity, 
whose advocates clung to an ancient and unknown stratum, because 
they believed that thence precisely would flow the healing serum 
of the nation that was to immunize it and help it to survive in 
ti^es of change and crisis, as was taking place at the present 
time. 

These two examples, extreme in their essence, illustrate the 
internal struggles among the literary groups. While for the 
first group separation had been a conscious act, latent trends 
toward continuity operated within the latter group, marking its 
Zionistic-pioneering nature. From the beginning this was a 
separation for the sake of continuity. This distancing 
themselves from the national heritage was not planned purposely, 
but was the result of major changes in the life of the Jewish 
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people — of the descent of one way of life and cultural patterns 
on the one hand, and the rise of another on the other hand. In 
any case, the goal of continuity was latent during the period of 
uprootadness , and conscious separation, which the pioneering 
ancestors "had bequeathed their sons, did not characterize the 
atrophic and spoiled patterns of existence in axile. 



Scoring Key for "Problems with New Israeli Prose: Between 
Isolation and Integration" 

1. There are two trends in modern Hebrew prose: first, 
separatism from the literary heritage and later on return to this 
heritage. 

2. Separatism from the literary heritage was not created 
purposely and consciously but rather was a temporary and 
unavoidable result of ignorance of the heritage. 

3. Despite attempts at separatism of modern Hebrew prose from 
its heritage, the younger generation started to recognize the 
value of the literary heritage and its spiritual richness. 

4. The author gives two extreme examples of the separatism that 
lead ultimately to a coming to terms with this heritage. 

5. First example: th& Yevreyskaya in the Soviet Union. 

6. Second example: the "Canaanite" group in Israel. 

7. The concluding sentence concerning the trends of continuity 
within the separatism and/or the continuity as a need within 
separatism in new Israeli prose. 



EFL Texts and Scoring Keys 

HOW TO AVOID FOOLISH OPINIONS 
by Sertrand Russell 

To avoid the various foolish opinions to which mankind is 
prone, no superhuman genius is required. A few simple rules will 
keep you, not from all error, but from silly error. 

If the matter is one that can be settled by observation, 
make the observation yourself. Aristotle could have avoided the 
mistake of thinking that women have fewer teeth than men, by the 
simple device of asking Mrs. Aristotle to keep her mouth open 
while he counted. He did not do so because he thought he knew. 
Thinking that you know when in fact you don't is a fatal mistake, 
to which we are all prone. I believe myself that hedgehogs eat 
black beetles, because I have been told that they do; but if I 
were writing a book on the habits of hedgehogs, I should not 
commit myself until I had seen one enjoying this unappetizing 
diet. Aristotle, however, was less cautious. Ancient and 
medieval authors know all about unicorns and salamanders; not one 
of them thought it necessary to avoid dogmatic statements about 
them because he had never seen one of them. 

Many matters, however, are less easily brought to the test 
of experience, if, like most of mankind, you have passionate 
convictions of many such matters, there are ways in which you can 
make yourself aware of your own bias. If an opinion contrary to 
your own makes you angry, that is a sign that you are 
subconsciously aware of having no good reason for thinking as you 
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do. If someone maintains that two and two are five, or that 
Iceland is on the Equator, you feel pity rather than anger, 
unless you know so little of arithmetic or geography that his 
opinion shakes your own contrary conviction. The most savage 
controversies are those about matters as to which there is no 
good evidence either way. Persecution is used in theology, not 
in arithmetic, because in arithmetic there is knowledge, but in 
theology there is only opinion. So whenever you find yourself 
getting angry about a difference of opinion, be on your guard; 
you will probably find, on examination, that your belief is going 
beyond what the evidence warrants. 

k good way of ridding yourself of certain kinds of dogmatism 
is to become aware of opinions held in social circles different 
from your own. When I was young, I lived much outside my own 
country — in France, Germany, Italy, and the United States. I 
found this very profitable in diminishing the intensity of 
insular prejudice. If you cannot travel, seek out people with 
whom you disagree, and read a newspaper belonging to a party that 
is not yours. If the people and the newspaper seem mad, 
perverse, and wicked, remind yourself that you seem so to them. 
In this opinion both parties may be right, but they cannot both 
be wrong. This reflection should generate a certain caution. 

For those who have enough psychological imagination, it is a 
good plan to imagine an argument with a person having a different 
bias. This has one advantage, and only one, as compared with 
actual conversation with opponents; this one advantage is that 
the method is not subject to the same limitations of time and 
space. Mahatma Gandhi deplored railways and steamboats and 
machinery; he would have liked to undo the whole of the 
industrial revolution. You may never have an opportunity of 
actually meeting anyone who holds this opinion, because in 
Western countries most people take the advantages of modern 
technique for granted. But if you want to make sure that you are 
right in agreeing with the prevailing opinion, you will find it a 
good plan to test the arguments that occur to you by considering 
what Gandhi might have said in refutation of them. I have 
sometimes been led actually to change my mind as a result of this 
kind of imaginary dialogue, and, short of this, I have frequently 
found myself growing less dogmatic and cocksure through realizing 
the possible reasonableness of a hypothetical opponent. 

Be very wary of opinions that flatter your self-esteem. 
Both men and women, nine times out of ten, are firmly convinced 
of the superior excellence of their own sex. There is abundant 
evidence on both sides. If you are a woman, you can retort that 
so are most criminals. The question is inherently insoluble, but 
self-esteem conceals this from most people. We are all, whatever 
part of the world we come from, persuaded that our own nation is 
superior to all others. Seeing that each nation has its 
characteristic merits possessed by our nation are the really 
important ones, while its demerits are comparatively trivial. 
Here, again, the rational man will admit that the question is one 
to which there is no demonstrably right answer. It is more 
difficult to deal with the self-esteem of man as man, because we 
cannot argue out the matter with some non-human mind. The only 
way I know of dealing with this general human conceit is to 
remind ourselves that man is a brief episode in the life of a 
small planet in a little corner of the universe, and that for 
aught we know, other parts of the cosmos may contain beings as 
superior to ourselves as we are to jelly-fish. 
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Scoring Key for "How to Avoid Foolish Opinions" 

1. The article suggests procedures for avoiding foolish 
opinions. 

2- First, observe things for yourself whenever possible. 

3. Second, if observation is out, discover any biases you may 
have about an issue. /because belief might go beyond evidence. 

4. Rid yourself of prejudice by discovering the opinions held by 
others — whether through meeting people out of your social 
group, travel, or through the newspaper. 

5. Another means is to nave an imaginary argument with an 
hypothetical opponent. 

6. Finally, be wary of taking a position which makes you right 
at the expense of the opposite sex or another nation since it is 
a no-win situation. /Be wary of opinions that flatter your self- 
esteem . 

Linking : intro that procedures/rules for avoiding foolish 
opinions/dogmatism . 



MODERN CONSTITUTIONS 

by K.C. Whears 

Most countries make the Constitution superior to the 
ordinary law. 

If we investigate the origins of modern Constitutions , we 
find that, practically without exception, they were drawn up and 
adopted because people wished to make a fresh start, so far as 
the statement of their system of government was concerned. The 
desire or need for a fresh start arose either because, as in the 
United States, some neighboring communities wished to unite 
together under a new government, or because, as in Austria and 
Hungary or Czechoslovakia after 1918, communities had been 
released from an Empire as the result of a war and were now free 
to govern themselves; or because a revolution had made a break 
with the past and a new form of government on new principles was 
desired? or because defeat in war had broken the continuity of 
government and a fresh start was needed after the war. The 
circumstances in which a break with the past and the need for a 
fresh start caorae about vary from country to country, but in 
almost every case in modern times, counties have a Constitution 
for the very simple and elementary reason that Uiey wanted, for 
some reason, to begin again and so they put down in writing the 
main outline, at least, of their proposed system of government. 
This has been the practice certainly since 1781 when the American 
Constitution was drafted, and as the years passed, no doubt 
imitation and the force of example have led all countries to 
think it necessary to have a Constitution. 

This does not explain, however, why many countries think it 
necessary to give the Constitution a higher status in law than 
other rules of law. The short explanation of this phenonmenon is 
that in many countries a Constitution is thought of as an 
instrument by which government can be controlled. Constitutions 
spring from a belief in limited government. Countries differ, 
however, in the extent to which they wish to impose limitations. 
Sometimes the Constitution limits the legislature also, but only 
so far as amendment of the Constitution itself is concerned; and 
sometimes it imposes restrictions upon the legislature which go 
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far beyond this point and forbid it to make laws upon certain 
subjects or in a certain way or with certain effects. Whatever 
the nature and the extent of the restrictions, however, they are 
based on a common belief in limited government and in the use of 
a constitution to impose these limitations. 

The nature of the limitations to be imposed on a government, 
and therefore the degree to which a Constitution will be supreme 
over a government, depends upon the objects which the framers of 
the Constitution wish to safeguard. In the first place they may 
want to do no more than ensure that the Constitution is not 
altered casually or carelessly or by subterfuge or by 
implication; they may want to secrue that this important document 
is not lightly tampered with, but solemnly, with due notice and 
deliberation, consciously amended. In that case it is legitimate 
to require some special ^ocess of constitutional amendment — 
say, that the legislature may amend the Constitution only by a 
two-thirds majority or after a general election or perhaps upon 
three months notice. 

The framers of Constitutions have more than this in mind. 
They may feel that a certain kind of relationship between 
legistlators and the executive is important, or that the 
judicatures should have a certain guaranteed degree of 
independence of the legislature and executive. They may feel 
that there are certain rights which citizens have and which the 
legislature or the executive must not invade or remove. They may 
feel that certain laws should not be made at all. The framers of 
the American Constitution, for example, forbade Congress to pass 
an ex-post facto law, that is, a law made after the occurence of 
the action or the situation which it seeks to regulate — a type 
of law which may render a man guilty of an offence through an 
action which, when he committed it, was innocent. The framers of 
the Irish Constitution of 1937 forbade the legislature to pass 
any law permitting divorce. 

Further safeguards may be called for when distinct and 
different communities decide to join together undtr a common 
government but are anxious to retain certain rights for 
themselves. If these communities differ in language, race, and 
religion, safeguards may be needed to guarantee to them a free 
exercise of these national characteristics. Those who framed the 
Swiss, the Canadian, and the South African Constitutions, to name 
a few only, had to consider these questions. Even when 
communities do not differ in language, race, or religion, they 
may still be unwilling to unite unless they are guaranteed a 
ameasure of independence inside the union. To meet this demand 
the Constituion must not only divide powers between the 
government of the Union and the governments of the individual 
component parts, but it must also be supreme in so far at any 
rate as it enshrines and safeguards this division of powers. 

In some countries only one of the considerations mentioned 
above may operate; in others some, and in some, all. Thus, in 
the Irish Constitution, the framers were anxious that amendment 
should be a deliberate process, that the rights of citizens 
should be safeguarded, and that certain types of laws should not 
be passed at all, and therefore, they made the Constitution 
supreme and imposed restrictions upon the legislature to achieve 
these ends. The framers of the American Constitution also had 
these objects in mind, but on top of that they had to provide for 
the desire of the thirteen colonies to be united for some 
purposes only and to remain independent for others. This was an p 
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additional reason for giving supremacy to the Constitution and 
for introducting certain extra safeguards into it. 



Scoring Key for "Modem Consitutions" 

1. The text first gives the historical reason for constitutions 
— namely, that people wanted to have a fresh start at their 
system of government. 

2. Usually the constitution was put above other laws that the 
legislature or executive might wish to instate. /supremacy of 
Constitution to control government. 

3. This power was granted the constitution to guard against 
efforts to tamper with the constitutional process. 

4. Thus, constitutions often include rights (of citizens) which 
cannot be invaded or removed by the government. /to limit the 
legislature as to amendment of the Constitution. /to limit the 
powers of government. 

5. There may also be stipulations protecting the rights of 
special communities governed by the same constitution, 
[constituent states/colonies in the case of the US] 

6. It is noted that countries differ as to the number of special 
checks and balances stipulated by their constitution. 

Linking : Opening topic sentence about Constitution serving as 
fresh start ./Degree to which the Constitution limits government 
depends on. . . 



SPECIALIZATION 
by Ortega y Gasset 

Specialization commences precisely at a period which gives 
civilized man tne title "encyclopedic." The intellectual history 
of the nineteenth century starts on its course under the 
direction of beings who live "encyclopedically , » though their 
production already has some tinge of specialism. in the 
following generation, the balance is upset and specialism begins 
to dislodge culture from the individual scientist. When, by 
1890, a third generation assumes intellectual command in Europe 
we meet with a type of scientist unparalleled in history. He is 
one who, out of all that has to be known in order to be a man of 
judgement, is only acguainted with one science — and, even of 
that one, he only knows the small corner in which he is an active 
investigator. He even proclaims it as a virture that he takes no 
cognisance of what lies outside the narrow territory specially 
cultivated by himself and gives the name of "dilettantism" to any 
curiosity for the general scheme of knowledge. 

What happens is that, enclosed within the narrow limits of 
his visual field, he does actually succeed in discovering new 
facts and in advancing the progress of the science which he 
hardly knows — and, incidentally, the encyclopedia of thought of 
which he is conscientiously ignorant. How has such a thing been 
possible, how is it still possible? For it is necessary to 
insist upon this extraordinary but undeniable fact: experimental 
science has progressed thanks in great part to the work of men 
astoundingly mediocre and even less than mediocre. That is to 
say, modern science, the root and symbol of our actual 
civilization, finds a place for the intellectually commonplace 
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man and allows his to work therein with success. The reason for 
this lies in what is at the sane time the great advantage and the 
gravest peril of the new science , and of the civilization 
directed and represented by it, namely, mechanization. A fair 
amount of the things that have to be done in physics or in 
biology is mechanical work of the mind which can be done by 
anyone, or almost anyone. For the purpose of innumerable 
investigations it is possible to divide science into small 
sections, to enclose oneself in one of these, and to leave out of 
consideratioii all the rest. The solidity and exactitude of the 
methods allow for this temporary but quite real dismemberment of 
knowledge. The work is done under one of these methods as with a 
machine and, in order to obtain quite abundant results, it is not 
even necessary to have rigorous notions of their meaning and 
foundations. In this way the majority of scientists help the 
general advance of science while shut up in the narrow cell of 
their laboratory, like the bee in the cell of the hive. 

But this creates an extraordinarily strange type of man. 
The investigator who has discovered a new fact of nature must 
necessarily experience a feeling of power and self assurance. 
With a certain apparent justice, he will look upon himself as "a 
man who knows." And, in fact, there is in him a portion of ^ 
something which, added to many other portions not existing in 
him, does really constitute knowledge. This is the true inner 
nature of the specialist who, in the first years of this ^ century , 
has reached the wildest stage of exaggeration. The specialist 
"knows" very well his own, tiny corner of the universe; he is 
radically ignorant of all the rest. 

Here we have a precise example of this strange new man, whom 
T have attempted to define, from both of these two opposite 
aspects. I have said that he was a human product unparalleled in 
history. The specialist serves as a striking, concrete example 
of the species, making clear to us that radical nature of the 
novelty. For, previously, a man could be divided simply into the 
learned and the ignorant — those who fit the former category and 
those that fit the latter. But our specialist cannot be brought 
in under either of these two categories. He is not lerned, for 
he is formally ignorant of all that does not enter into his 
specialty; but neither is he ignorant, because he is "a 
scientist," and "knows" very well his own tiny portino of the 
universe. We shall have to say that he is a learned ignoramus — 
which is a very serious matter, as it implies that he is a person 
who is ignorant, not in the fashion of the ignorant may, but with 
all the petulance of one who is learned in his own special line. 

Scoring Key for "Specialization" 

1. By the end of the nineteenth century the intellectual 
generalists had given way to specialists, people versed only in 
one science rather than having a broad knowledge base, (during 
the 19th C.) [not necessarily contrasting with intellectual 
generalists ] 

2. While these scientists have made progress in their fields, 
they remain ignorant of the rest of the world. 

3. Even in their fields, mechanization has allowed them to do 
successful work on the basis of a minimum of knowledge. 

4. The author sees the situation to be producing strange new 
people: specialists who are not learned, yet are not ignorant in 
that they are scientists, hence "learned ignoramuses." 



